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Abstract 

Background: Our goal is to validate the Memorial Sloan-Kettering Cancer Center (MSKCC) nomogram and Stanford Online 
Calculator (SOC) for predicting non-sentinel lymph node (NSLN) metastasis in Chinese patients, and develop a new model 
for better prediction of NSLN metastasis. 

Methods: The MSKCC nomogram and SOC were used to calculate the probability of NSLN metastasis in 120 breast cancer 
patients. Univariate and multivariate analyses were performed to evaluate the relationship between NSLN metastasis and 
clinicopathologic factors, using the medical records of the first 80 breast cancer patients. A new model predicting NSLN 
metastasis was developed from the 80 patients. 

Results: lV\e MSKCC and SOC predicted NSLN metastasis in a series of 120 patients with an area under the receiver 
operating characteristic curve (AUC) of 0.688 and 0.734, respectively. For predicted probability cut-off points of 10%, the 
false-negative (FN) rates of MSKCC and SOC were both 4.4%, and the negative predictive value (NPV) 75.0% and 90.0%, 
respectively. Tumor size, Kiss-1 expression in positive SLN and size of SLN metastasis were independently associated with 
NSLN metastasis (p<0.05). A new model (Peking University People's Hospital, PKUPH) was developed using these three 
variables. The MSKCC, SOC and PKUPH predicted NSLN metastasis in the second 40 patients from the 120 patients with an 
AUC of 0.624, 0.679 and 0.795, respectively. 

Conclusion: MSKCC nomogram and SOC did not perform as well as their original researches in Chinese patients. As a new 
predictor, Kiss-1 expression in positive SLN correlated independently with NSLN metastasis strongly. PKUPH model achieved 
higher accuracy than MSKCC and SOC in predicting NSLN metastasis in Chinese patients. 
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introduction 

Sentinel lymph node biopsy (SLNB) has taken the place of 
axillary lymph node dissection (ALND) in early stage breast 
carcinoma and its benefits in terms of morbidity have been well 
estabhshed [1]. But for breast cancer patients with sentinel lymph 
node (SLN) metastases, the benefit of ALND on survival is 
debated. The American College of Surgeons Oncology Group 
(ACOSOG) ZOO 11 trial showed that women with Tl and T2 
tumors who undergo lumpectomy derive little additional benefit 
from ALND since any residual disease in the level I and II nodes 
appear to be effectively eradicated by postoperative irradiation 
and chemotherapy [2]. While in most centers, the current 
standard of treatment remains ALND [3]. However, 40% to 
60% of patients have no disease in axillary lymph nodes other than 
the SLN itself and this means these patients undergo unnecessary 
ALND [4—6]. Therefore, the ability of a diagnostic test to predict 



the chance of NSLN involvement is critical to avoid unnecessary 
ALND. 

In the past 10 years, multiple models have been proposed to aid 
in the stratification of a patient's risk of having NSLN metastasis 
[7-10], in which the nomogram developed at the Memorial Sloan- 
Kettering Cancer Center [1 1] and the model of Stanford Online 
Calculator (SOC) [12] are the most well-validated and widely 
used. However, both models were estabhshed based on a Western 
population. It is well known that there are some difference in the 
clinical characteristics and treatment modes between Chinese 
breast cancers patients and Westerners. Therefore, there is a clear 
need of studies to evaluate these nomograms in Chinese breast 
cancer patients. 

To date, all the models predicting NSLN status are based on the 
routine histopathological variables of the primary tumor and its 
metastasis [13,14]. It is uncertain if there are some additional 
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Figure 1 . Area under the receiver operating characteristic curve (AUC) for iVISKCC and SOC models(n = 1 20). Diagonal line represents 
an AUC of 0.5, indicating a score equal to chance. 
doi:1 0.1 371 /journal.pone.01 041 1 7.g001 



biological markers that may be able to predict the risk of NSLN 
metastasis. 

In the current study, we validated MSKCC and SOC models in 
predicting NSLN metastasis in Chinese breast cancer patients. In 
addition, we identified new factors that may predict NSLN 
metastasis and developed a novel predictive model specifically for 
Chinese patient population. 

Materials and Methods 

Study patients 

We reviewed 120 consecutive, clinically lymph node negative 
breast cancer patients with positive SLN(s) who underwent 
completion ALND from January 2009 through December 2012 
at the Breast Disease Center, Peking University People's Hospital 
(PKUPH). Approval from PKUPH's review board was obtained 
before data collection. Written consent has been obtained from all 
the patients. 

Surgery and SLN pathological evaluation 

SLNs were identified using fluorescence and/or blue dye 
according to surgeon preference. Intraoperative frozen section 
was performed on all SLNs. The SLN was cut longitudinally into 2 
halves. Half of the node was frozen for immediate examination, 
and up to 2 sections were stained with hematoxylin and eosin 
(H&E). The other half node was fixed in formalin and embedded 
in paralfin, and up to 2 sections were stained with Hc&E. 
Immunohistochemical stain was not routinely used in the diagnosis 
of SLN metastasis. 



The size of metastatic tumor deposits within all nodes was 
categorized according to the American Joint Committee on 
Cancer Staging Manual (7th edition) as follows: isolated tumor 
cells (ITC) were defined as tumor deposit of 0.2 mm or less (pNOi+ 
), micrometastases (MI) were defined as tumor deposit of more 
than 0.2 mm but not more than 2 mm (pNlmi), and macro- 
metastases were defined as tumor deposit of more than 2 mm 
(pNl). 

Axillary dissection was performed if SLN was positive by frozen 
section analysis. Patients with SLN metastases who were not 
detected during operation generally underwent completion ALND 
at a later date. For all additional nodes identified by completion 
ALND, routine H&E analysis was conducted on a single section of 
each node. 

IVISKCC nomogram and SOC 

The MSKCC nomogram is based on 9 histopathologic 
variables: primary tumor size (in centimeters), primary tumor 
type (ductal/lobular) and nuclear grade (1-3), number of positive 
SLNs, number of negative SLNs, method of SLN detection 
(frozen, routine H&E, serial H&E, or immunohistochemistr 
[IHC]), lymphovascular invasion (LVI) (yes/no), multifocality 
(yes/no) and estrogen receptor positivity (yes/no). 

The Stanford nomogram requires 3 histopathologic variables: 
primary tumor size (in centimeters), size of SLN metastasis (ITC, 
MI, and macrometastases) and LVI (yes/no). 
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Clinical and pathological characters 

Clinical and pathological data collected for each case included 
age, tumor size (pathological size of the invasive carcinoma), 
tumor site (superior internal, inferior internal, superior external, 
inferior external and central quardrant), tumor type (ductal or 
lobular carcinoma, ductal and lobular mixed carcinoma, special 
type carcinoma (such as mucinous adenocarinoma and so on), 
nuclear grade, multifocality, presence of LVI, molecular subtype, 
expression of estrogen receptor (ER), progesterone receptor (PR), 
Her-2, Ki67, CK5/6, p53, EGFR, E-cadherin, Kiss-1, mn23 m 
tumor, expression of Kiss-1 and nm23 in positive SLNs, reduction 
of Kiss- 1 expression from tumor to positive SLN (Kiss- 1 positive in 
breast tumor while negative in positive SLN), reduction of nm23 
expression from tumor to positive SLN (nm23 positive in breast 
tumor while negative in positive SLN), method of detection of 
SLN metastases (frozen-section analysis [frozen], routine H&E 
[routine H&E], H&E stains of serial sections [serial HE]), number 
of positive SLN, number of negative SLN, number of total SLN, 
ratio between positive SLNs and total amount of SLN, size of SLN 
metastasis (ITC, MI, and macrometastases), and extracapsular 
invasion (ECI) at positive SLNs, ratio of SLN metastasis size and 
positive SLN size with total of 30 factors. 

Statistical analysis 

We used the 120 patients' data to validate the performance of 
MSKCC and SOC nomogram. Model discrimination was 
assessed by calculating the area under the receiver operating 
characteristic curve (AUG) with 95% confidence interval (95% 
CI). The nomograms were further assessed the ability to accurately 
identify patients at very low risk for NSLN metastasis using false- 
negative rate (FN), negative predictive value (NPV), sensitivity, 
specificity and overall predictive accuracy with the cutoff value of 
10%. 

We tried to develop a novel predictive model specifically for 
Chinese patients according to the records of the first 80 
consecutive patients from the 120 patients. First, univariate 
analysis of the 30 factors described above was performed to 
determine which one was associated with NSLN metastases. Chi- 
squared test and Fisher's exact test were used for categorical 
variables and Mann-Whitney U test for continuous variables. The 
predictors with p value less than 0.05 were included in the 
multivariate analysis. All variables with p value<0.05 were 
considered statistically significant in the multivariate analysis and 
were then included into a logistic regression analysis using 
backward stepwise method to create the final predictive model. 

The resulting multivariate predictive model was then validated 
with the 80 series themselves. Discrimination of the model was 
assessed using AUC and the Hosmer-Lemeshow goodness-of-fit 
test. AUC, false-negative rate, negative predictive value, sensitiv- 
ity, specificity and overall predictive accuracy with the cutolf value 
of 10% were compared between the new model, MSKCC model 
and SOC model using the sample of additional 40 patients from 
the 120 patients. 

SPSS 16.0 software was used for statistical analyses. 

Results 

Forty five (37.5%) of the 120 patients from PKUPH had NSLN 
metastasis and the descriptive tumor and nodal characteristics of 
this cohort used in the MSKCC and Stanford nomograms are 
listed in Table 1. Compared with the MSKCC [1 1] and Stanford 
studies [12], difference was found in age, tumor type, nuclear 
grade, LVI, multifocality, method of SLN detection and size of 
positive SLN metastasis. The ROC curves generated by MSKCC 
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and Stanford nomograms are shown in Figure 1 . The AUG value 
of the Stanford and MSKCC models was 0.734(95% CI, 0.644- 
0.825) and 0.688(95% CI, 0.589-0.787), respectively. 

When patients with a low risk of having NSLN disease were 
examined, we found that, based on the MSKCC nomogram, 8 
patients in this study cohort had a 10% predicted probability of 
having NSLN metastasis. Of these 8 patients, 2 had positive 
NSLNS with a FN rate of 4.4% and NPV of 75.0%. The 
sensitivity and specificity was 95.6% and 8.0%, respectively, with 
an overall predictive accuracy of 40.8%. Compared with MSKCC 
nomogram, 20 patients by the SOC model were predicted to have 
a 10% probability of having NSLN metastasis, of whom 2 had 
positive NSLNS with a FN rate of 4.4% and NPV of 90.0%. The 
sensitivity and specificity was 95.6% and 24.0%, respectively, with 
an overall predictive accuracy of 50.8% (Table 2). 

Univariate analysis of the patients with additional metastases on 
NSLN and those with no additional metastases on NSLN are 
given in Table 3. Significant difference was found between the two 
groups in the tumor size, tumor type, LVI, tumor p53 expression, 
expression of kiss- 1 in positive SLN, decrease of kiss- 1 expression 
from tumor to positive SLN, number of positive SLN, size of SLN 
metastasis, the ratio of SLN metastasis size and positive SLN. 
Expression of Kiss- 1 in positive SLN is correlated with decrease of 
Kiss-1 expression from tumor to positive SLN. Size of SLN 
metastasis is correlated with the ratio of SLN metastasis size and 
positive SLN size. To avoid the interaction of these predictors, 
decrease of Kiss- 1 expression from tumor to positive SLN and the 
ratio of SLN metastasis size with positive SLN size were not 
included in the multivariate analysis. 

By multivariate analysis, tumor size, expression of Kiss-1 in 
positive SLN and the size of SLN metastasis remained significantly 
predictive of NSLN status (p<0.05). The results of the backward 
stepwise binary logistic regression analysis are given in Table 4. 
The multivariate logistic regression analysis produced the follow- 
ing mathematical predictive model (PKUPH model) for NSLN 
status in our patient cohort, with the Y' denoting the probability of 
NSLN metastases: 

Y' = 

£XP[- 3.901 +0.841 X (X3) -1.831 x (X21) + 2.909 x (X29)] 
I+£-jrP[-3.90I+0.841 x(X3)-I.831 x (X2I) + 2.909 x (X29)] 

Goodness-of-fit of the model was assessed by Hosmer-Lemeshow 
(HL) test that produced a p value of 0.967, indicating that the 
multivariate model fits well for the patient population. This new 
model's discrimination was compared with the MSKCC and SOC 
models using the 80 patients themselves. The AUC value of the 
PKUPH model was 0.854 (95% CI, 0.772-0.936) while the AUC 
value of the MSKCC and Stanford model was 0.739 (95% CI, 
0.627-0.851) and 0.753 (95% CI, 0.639-0.866), respectively 
(Figure 2). 

The new model was further validated using the additional 40 
patients. The AUC value of the PKUPH model was 0.795 (95% 
CI, 0.651-0.940) that was superior to the AUCs of 0.624 (95% CI, 
0.443-0.804) and 0.679(95% CI, 0.514-0.845), generated by 
MSKCC and Stanford nomograms, respectively (Figure 3). The 
ability of these three models predicting patients with a low 
probability of non-SLN metastases was further compared with a 
cutoff value of 10%. The PKUPH model had a FN rate of 5.6%, 
NPV of 92.3%, specificity of 54.5%, sensitivity of 94.4%, and 
overall predictive accuracy of 72.5%. The FN rate, NPV, 
specificity, sensitivity and overall predictive accuracy of MSKCC 
model were 11.1%, 50%, 9.1%, 88.9% and 45%, respectively. 
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Figure 2. Area under the receiver operating characteristic curve (AUC) for MSKCC, SOC, and PKUPH models (n = 80). 

doi:1 0.1 371 /journal.pone.01 041 1 7.g002 



The FN rate, NPV, specificity, sensitivity and overall predictive 
accuracy produced by SOC model was 5.6%, 80%, 18.2%, 94.4% 
and 52.5% (Table 5). 

Discussion 

With the adoption of SLN biopsy, a new clinical conundrum 
has become commonplace: should a completion ALND be 
performed for a patient with a positive SLN biopsy? It is difficult 
to accurately estimate the risk of NSLN metastases for an 
individual breast cancer patient using the present methods, so 



ALND has been considered the golden standard for breast cancer 
patients with SLN metastasis. 

Some tumor and nodal characteristics have been identified as 
risk factors for the presence of NSLN metastasis [13,15-17]. The 
presence of predictive model provides a good way simultaneously 
including several variables that are acquired through mathemat- 
ical method in a large population. The MSKCC model was the 
first nomogram to predict NSLN metastasis following a positive 
SLN biopsy [11], which produced an AUC of 0.72 versus the 
clinicians' AUC of 0.54 [18]. Subsequent validation studies have 



Table 4. Multivariable logistic regression of clinicopathologic data and NSLN involvement (n = 80). 





Factor 


Characteristic 


fi 


S.E 


P 


OR 


95% a 


X3 


Tumor size 


0.904 


0.449 


0.044 


2A7 


1 .024-5.952 


X4 


Tumor typr 


0.363 


0.404 


0.368 


1.44 


0.652-3.171 


X6 


LVI 


1.280 


1.361 


0.347 


3.60 


0.249-51.833 


XI 5 


p53 


1.289 


0.719 


0.073 


3.63 


0.887-14.860 


X21 


Kiss-l(SLN) 


-1.791 


0.686 


0.009 


0.17 


0.043-0.641 


X24 


No. of positive SLNs 


0.409 


0.411 


0.321 


1.50 


0.672-3.370 


X29 


Size of SLN metastasis 


2.371 


0.895 


0.008 


10.71 


1.853-61.907 




Constant 


-5.469 


1.629 


0.001 


0.004 





S.E: Standard error. 

OR- Odds ratio. 

CI: Confidence intervals. 

doi:l 0.1 371/journal.pone.Ol 041 1 7.t004 
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Figure 3. Area under the receiver operating characteristic curve (AUC) for MSKCC, SOC, and PKUPH models (n = 40). 
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shown an AUC ranging from 0.58 to 0.78 in other centers [19- 
21]. In 2008, a modified nomogram using only 3 histopathologic 
variables was developed at Stanford [12]. The SOC has 
demonstrated an AUC of 0.64 to 0.73 by validation that seems 
to perform better than the MSKCC nomogram [22-24]. 

Predictive models are known to work best in the centers where 
they were developed and therefore require validation in other 
independent patient populations before being adapted to clinical 
use. We validated these two models in 120 Chinese breast cancer 
patients. Our result showed that the AUC generated by MSKCC 
and SOC nomogram were 0.688 and 0.734, respectively, which 
are inferior to their original AUC values(0.77 and 0.74). A NSLN 
predictive model is to select patients at low risk with NSLN 
involved mainly, but both MSKCC and SOC models showed 
poor clinical utility in Chinese breast cancer patients by selecting 
only 8 (6.7%) and 20 (16.7%) patients when a 10% cut-ofiF value 
was used. The specificity of these two models was only 8.0% and 
24%, respectively, which was inferior to most other validation 
studies in Western cancer centers. 

The different validation results between our center and Western 
ones might be due to the different tumor characteristics between 
Chinese and Western countries population (Table 1). In China, 
the number of pre-menopausal breast cancer patients is usually 
larger than that of Western countries [25]. For MSKCC 
nomogram, tumor type is a predictive factor only including ductal 
and lobular tumor two types while there are 31.7% patients having 
ductal and lobular mixed tumors in our study population. LVI has 
been found to be correlated strongly with NSLN positivity by some 
authors [11,12]. So both MSKCC and SOC models include LVI 
as a predictor. While the rate of LVI is 11.7% in our patient 



population which is much lower than the 40.5% in the MSKCC 
patients and 33.3% in SOC patients. For the diagnosis method of 
SLN metastasis, serial H&E and IHC have been widely used in 
Western cancer centers, while these are rarely used in China. It is 
clear that IHC is more sensitive than H&E in detecting 
micrometastases and that routine H&E analysis is more sensitive 
than frozen-section analysis. There is a correlation between 
method of detection and volume of tumor [1 1] and it has also 
been indicated that method of detection was correlated with the 
size of the SLN metastasis [26,27]. We found tiie number of ITC 
and micrometastasis in our 120 patients is 0% and 36.7%, 
respectively, which is much lower than those in Stanford and other 
studies. Given the dramatic difference exists between Chinese 
patients and Westerners, a predictive model based on Chinese 
population is in great need. 

Some histopathological variables of the primary tumor and its 
metastasis have been identified to correlate with the NSLN status. 
While, the estimates of risk for any given characteristic vary 
considerably among studies. Up to now, the size for primary 
tumor, grade of primary tumor, the maximum size of positive 
SLNs, LVI, extracapsular invasion in SLN, ER, PR and HER-2 
status were the mostly analyzed risk factors in other studies. In 
addition to these factors, we collected biomarkers as Ki67, CK5/6, 
p53, EGFR, E-cadherin, Kiss-l, nm23 to analyze their relation- 
ship with NSLN involvement. By univariate analyses, we found 
that 9 variables were correlated to NSLN metastasis: tumor size, 
tumor type, LVI, expression of p53 in tumor, expression of Kiss-l 
in positive SLNs, decrease of kiss-l expression from tumor to 
positive SLN, number of positive SLN, size of SLN metastasis, and 
the ratio of SLN metastasis size and positive SLN size. In the 
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multivariable logistic regression analysis, tumor size, expression of 
kiss- 1 in positive SLN and the size of SLN metastasis were each 
associated with the likelihood of NSLN metastases. We adopted 
these three factors to develop our new model to predict the 
probability of NSLN metastasis. 

Tumor size is an important factor influencing the lymph node 
involvement in breast cancer. Increasing tumor diameter has been 
shown to increase the overall risk of metastatic axillary lymph 
nodes and therefore the likelihood of tumor-positive SLNB [28]. It 
seems reasonable that risk factors for axillary lymph node 
metastases in general turn out to be risk factors for nonsentinel 
metastases after tumor-positive SLNB. This fact has been 
confirmed in some studies. Hwang et al reported no NSLN 
metastases in patients with Tla lesions whereas patients with T2, 
T3 and T4 tumors were associated with positive NSLNs in 54%, 
77% and 80% of cases, respectively [13]. Similar data were 
presented by Kamath [26], Joseph [16] and Chu [29] that a 
NSLN metastasis rate of 13% for Tib lesions, 38% for T2, and 
71% for T3 tumors. Both MSKCC and SOC models include 
tumor size as a predictive factor. In the current study, tumor size 
was found to be an independent risk factor for NSLN metastasis 
through univariate and multivariate analyses. Therefore we 
included tumor size in our new model. 

The size of SLN involvement has been identified as a significant 
predictor of NSLN metastases. Weiser et al. reported that patients 
with a metastasis size £2 mm in SLN have very low risk of NSLN 
metastasis [30] . Chu et al. [29] examined the SLNs and NSLNs of 
194 patients and found that 47% of the patients having positive 
SLNs with macrometastasis (^2 mm) could be identified NSLN 
metastasis only through H&E method. Consistently, other studies 
found that SLN metastasis size S2 mm was an independent 
predictor of NSLN metastasis [5], [13], [31]. Kohrt et al. [12] used 
the size of SLN metastasis as one of the three factors in the SOC 
model and many studies have validated this model with good 
results. While the MSKCC model didn't include the size of SLN 
metastasis. The authors did pointed out that the absence of size 
determination for the SLN metastases was a limitation of the 
MSKCC model [11]. In our study, we proved the size of SLN 
metastasis remained significantly predictive of NSLN status and 
adopted this variable in our new model. 

The current study is the first study to analyze the impact of 
tumor and positive SLN Kiss- 1 expression for NSLN involvement. 
Kiss-1 has been identified as a putative human metastasis 
suppressor gene in melanomas [32]. It has also been suggested 
as a potential metastasis suppressor in breast cancer cells without 
affecting tumorigenicity [33]. Mitchell and colleagues [34] found 
the loss of Kiss-1 gene expression in highly metastatic breast 
cancer cell lines. Stark et al [35] determined the expression of 
Kiss-1 in primary breast tumor and brain metastatic foci, and 
found that Kiss-1 expression in brain metastatic tumor was 10 
times lower than that in breast tumor, indicating that Kiss-1 gene 
might be involved in breast cancer metastasis. Kostadima et al 
[36] also have reported that the positive rate of Kiss-1 is only 3% 
in lymph node positive breast cancer, supporting the anti- 
metastatic role of the Kiss- 1 for breast cancer. Our previous study 
has indicated that Kiss-1 is the most important and independent 
impact factor for lymph node metastases in breast cancer [37]. In 
the current study, we found that patients with a reduction of Kiss-1 
expression in the positive SLN compared with the primary tumor 
have a higher chance for positive NSLN (p<0.05). Interestingly, 
primary breast cancer Kiss-1 expression was not correlated with 
NSLN metastasis. While Kiss-l positive in metastatic SLN was a 
strong negative predictive factor of NSLN involvement. Therefore, 
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we have included the SLN Kiss-1 expression in our predictive 
model. 

A good breast cancer NSLN metastasis predictive model 
depends on not only its predictive ability but also its clinical 
utility convenience. To our knowledge, to date, the majority of 
models with a low number of variables have performed inferiorly 
to models with more variables. While it is interesting that the SOC 
model with 3 variables is comparable to the MSKCC nomogram 
including 9 variables. Thus, factors with strong predictive strength 
may be more important than the number of variables for a model. 
Our new model has 3 variables among which the tumor size and 
the size of SLN metastasis are the same ones as SOC model. 
While, we have chosen Kiss- 1 in positive SLN instead of LVI for 
our model. As compared with MSKCC and SOC models, our 
new model performed better on AUC and ability to select patients 
at low risk of NSLN involvement at least in our center. 
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